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Abstract 

Check-In/Check-Out (CICO) is an effective and widely used Tier 2 behavior program in 
elementary and middle schools. This article describes a pilot study focused on examining the 
impact and usability of an adapted CICO program in a high school. We used a single-case 
multiple baseline design across five 9th grade students to evaluate the effects of the program on 
student academic engagement and disruption/non-compliance. In addition, we administered 
social validity questionnaires to school personnel and students to assess the usability of the 
program. Findings were mixed in terms of impact on student behaviors. Both school personnel 
and students found the program useable. Based on these findings, we discuss limitations and 
offer recommendations for how high school personnel could use CICO to support students at- 
risk of school failure. 
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Check-In/Check-Out in High Schools: An Exploratory Analysis 

Check-In/Check-Out (CICO; Crone, Hawken, & Horner, 2010) is a Tier 2 behavior 
program designed to support at-risk students. CICO has been evaluated in multiple empirical 
studies, and found to be both effective at improving student behavior (Campbell & Anderson, 
Miller, Dufrene, Sterling, Olmi, & Bachmeyer, 2015; Todd, Campbell, Meyer, & Homer, 2008). 
A recent synthesis of CICO research led the authors to identify the program as an “evidence- 
based practice” for improving social behavior of students in elementary and middle schools 
(Maggin, Zurheide, Pickett, & Baillie, 2015). 

Designed as a Tier 2 program, CICO is a common implemented within school-wide multi- 
tiered system of support framework, such as Positive Behavioral Interventions and Support 
(PBIS; Crone, Hawken & Horner, 2010). As a Tier 2 program, CICO is typically implemented as 
standardized program for all students identified as eligible for Tier 2 supports. The conceptual 
logic guiding CICO is to (a) increase the structure of a student’s day by adding pre-defined 
prompts for self-management, (b) provide explicit training in self-regulation skills such as self- 
monitoring, self-evaluation, self-delivered consequences, and self-recruited support, (c) use 
behavioral momentum to launch the school day and each class period with a positive interaction, 
(d) create opportunities for predictable and frequent adult recognition for appropriate behavior, 
(e) reduce inadvertent rewards contingent on problem behavior, (f) increase coordination of 
student support between home and school, and (g) collect and use daily data to adapt CICO 
elements to fit the needs of students (Crone, Hawken, & Horner, 2010; Hawken and Horner, 
2003). Five procedural components of CICO are designed to meet these conceptual goals: 

a) Daily Check-in/check-out: At the beginning and end of each day, the student briefly 


checks-in and out (e.g., 2-5 min) with an identified member of the school faculty/staff 
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b) 


c) 


d) 


(CICO Coordinator; e.g., counselors, teachers, behavioral specialists). The morning 
check-in allows an opportunity for the student to start each day with a positive 
interaction, and to pre-empt a potentially negative start to the day if a student arrives 
unprepared for school. The afternoon check-out with an adult allows a quick review of 
the day, and the opportunity to prompt positive behavior for the next school day. 

Daily point card: A daily point card is used in each class period throughout the day to 
monitor if the student meets expectations for social behavior. The daily point card lists 
school-wide expectations and has a place for each teacher to rate the student on each 
expectation as exhibited during their class period (e.g., 0 = did not meet expectations, 1 = 
partially met expectations, and 2 = met expectations). It also includes a point goal and a 
total of points available for the day (see Figure 1 for example). Many schools enter the 
daily points into the School-wide Information System (CICO-SWIS; May et al., 2016; 
https://www.pbisapps.org), a web-based data system for managing data entry and report 
generation on student performance in CICO. 

Self-regulation training: The CICO coordinator teaches each student behavioral 
expectations for the school, and the daily routine of CICO checks and card use. Emphasis 
is placed on student self-management and the skills needed for class by class success. 
Rating and feedback: At predetermined transitions (e.g., end of class periods), the 
student uses the daily point card to first self-assess their behavior, and then prompt their 
teachers to provide feedback on their performance. The teacher feedback includes verbal 
praise for behavioral expectations met during that time period and encouraging, 


corrective feedback if the student did not meet the expectations. 


CHECK-IN/CHECK-OUT IN HIGH SCHOOLS fs) 


e) Home-school communication: After daily check-outs, the option exists for the student to 
take the daily point card home for a parent/ guardian to sign. The student then returns the 
signed daily point card to school the following morning. Parents/guardians are 
encouraged to provide positive feedback to the student and not to establish any punitive 
contingencies for not meeting goals. This feature increases parent/guardian 
communication, awareness, and involvement, but is optional based on family and student 
preference (Crone et al., 2010; Hawken, Bundock, Kladis, O’Keeffe, & Barrett, 2014; 
Hawken and Horner, 2003; Wolfe et al., 2016). 

Although CICO is proven to be an effective program for students in elementary and 
middle schools, its application and effectiveness for students in high schools is uncertain. For 
example, due to the developmental age of high school students, and the organizational features 
and structures of high schools, it is more complex to implement behavior programs in these 
environments (e.g., larger class sizes, multiple buildings and teachers, increased emphasis placed 
on students to meet academic expectations; Flannery, Frank, Kato, Doren, & Fenning, 2013). In 
addition, Swain-Bradway (2009) also found certain adaptations to the traditional CICO program 
to be useful in high schools. For example, using a single-case multiple-baseline design across six 
high school students, the author evaluated the effects of traditional components of the CICO 
program, combined with an academic seminar-type class focused on increasing study skills, such 
as notebook organization, test taking strategies, goal setting (Swain-Bradway, 2009). Results 
indicated that after implementation of the adapted CICO program, there was an increase in 
student academic engagement. 

Moreover, in a recent survey study conducted by Kittelman, Monzalve, Flannery, & 


Hershfeldt (2018) provides some evidence that school personnel are focused on implementing 
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and adapting CICO in high schools. Of personnel surveyed from 18 high schools, the authors 
found that the majority had adapted at least some traditional procedural components of CICO. 
Adaptations included having students check-in and check-out with multiple school personnel and 
varying times of the day, providing feedback to family members electronically, and students 
receiving ratings and feedback from high school teachers based on behavior and academic 
expectations (Kittelman et al., 2018). The authors indicated that some adaptions were made 
because students were not buying into traditional procedural components of the CICO program 
(Kittelman et al., 2018). 

Based on these findings, it is plausible that adaptations to CICO are necessary in high 
schools for (a) students to participate in the program and (b) fit within the organizational 
structures and content of high schools. It might be possible to adapt certain procedural 
components of CICO, such as the daily point card, are helpful to increase daily student 
participation in the program. Therefore, there is a need for additional exploratory research in this 
domain to test and evaluate impact and usability of CICO in high schools. 

Present Study 

The purpose of this study was to further explore the usability and impact of CICO for 
high schools. We conducted an exploratory pilot study designed to evaluate whether an adapted 
CICO program for high schools could be implemented with high fidelity. We also sought to 
evaluate whether implementation of the adapted program would result in improve student 
outcomes (i.e., increased academic engagement and decreased disruption/non-compliance 
behaviors) and be rated useable in high schools by school personnel and students. The research 
questions for this pilot study were: 


1. To what extent can CICO-Secondary be implemented with fidelity? 
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2. Does implementation of CICO-Secondary result in improve student behaviors in high 
schools? 
3. Do school personnel and students find CICO-Secondary useable in high schools? 
Method 
Setting and Participants 

The participants were five ninth grade students, their teachers, and one school behavior 
specialist (CICO Coordinator) from a midsize, suburban high school in the Pacific Northwest. 
The study took place from April to June of the 2017-18 school year. According to school 
demographic data from the National Center for Education Statistics (NCES) database, the high 
school included a total of 1,403 students. The majority of students were identified as either 
White (57%) or Hispanic (30%). In addition, 860 (61%) were free lunch eligible and 118 (8%) 
were reduced-price lunch eligible. 

Students. Of the five students, three were males and two were females. Two male 
students were White and the other Black. Two of the males were 15 and the other 14 years old. 
Both female students were White and 15 years old. None of the students had IEPs. To be eligible 
to participate they had to have: (a) received multiple office discipline referrals during the school 
year, and/or (b) be nominated by a teacher, team, administrators, or behavior specialist due to 
frequent classroom problem behaviors (e.g., off-task, disruption). Similar inclusion criteria was 
used by Hawken and Horner (2003) when evaluating some of the initial effectiveness of the 
traditional CICO program. Since the high school operated with an alternating A day (periods 1, 
3, 5, 7) and B day (periods 2, 4, 6, 8) schedule, school personnel were asked to identify two 


classes (one A and one B day) in which each student struggled with academic engagement and/or 
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problem behavior. These classes were used for collection of direct observational data. Classes 
were 90 min long, expect on early release days when classes were 65 min. 

Andrew. The two classes that Andrew was observed in were Science 9 (A days) and 
Career and College Readiness (B days) classes. 

Bethany. The two classes that Bethany was observed in were English 9 (A days) and 
Career and College Readiness (B days). 

Colin. The two classes that Colin was observed in were English 9 (A days) and Algebra 1 
(B days). 

David. The two classes that David was observed in were Science 9 (A days) and English 
9 (B days) classes. During the first three observations in baseline on A days, David was observed 
in Career and College Readiness; however, his schedule was changed and the remaining 
observations on A days occurred in Science 9. 

Ellise. The two classes that Ellise was observed in were English 9 (A days) and Algebra 
1 (B days). 

Teachers. The participating teachers provided instruction in each of the identified classes 
for the students. Note that direct observation was limited to the two classes selected as most 
relevant for each student (one A day and one B day class), but students (and teachers) used the 
daily point card in all classes. A total of 24 teachers participated in the study. 

CICO coordinator. The CICO coordinator was a white, female doctoral student in 
educational leadership and policy, who also worked as a full time learning behavior specialist at 
the high school. The CICO coordinator had a total of 10 years of experience working in 


education. The CICO coordinator participated in the study approximately 30 min per day, and 
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did not have full-time equivalent (FTE) specifically dedicated toward implementing the adapted 
CICO program. 
Measures 

Academic engagement and disruption/non-compliance. To evaluate the impact of 
CICO-Secondary, direct observation data were collected on two student behaviors: academic 
engagement and disruption/noncompliance. During one academic period, 2-3 days per week, 
observations occurred during the first 15 min of class and were recorded using a 10-s whole 
interval recording protocol to assess the percentage of intervals students were academically 
engaged. Direct observations began 5 min after class started and lasted for 10 min. Students were 
recorded as being academically engaged if they were performing behaviors that matched 
classroom instruction for at least seven of the 10-s in an interval. Examples of academic 
engagement included: (a) body orientated toward teacher or activity, (b) asking or answering 
questions, and (c) following teacher requests. 

The second variable of interest was disruption/non-compliance and was measured using a 
10-s partial interval recording system. Disruption behaviors were defined as behaviors that were 
distracting or upsetting for teachers or peers (e.g., taking out of turn or interrupting, inappropriate 
physical contact). Non-compliance was defined as students failing to comply with a teacher’s 
requests within five seconds (e.g., ignoring a teacher’s request or not following directions). If 
students came late to class, but within the first 15 min of class, data collectors were instructed to 
begin conducting the student observations at that time. Observations occurred only on school 
days when normal instruction was scheduled (e.g. not when exams or tests were taking place). 

Interobserver agreement (IOA). A total of eight data collectors participated in the 


study. Data collectors included special education doctoral students, a school psychology masters 
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student, an undergraduate student, and a former general education teacher. The first author 
collected all IOA data (secondary data collector) throughout the study, and served as a primary 
data collector on limited occasions when other data collectors were not available. Data collectors 
were trained by the first author to collect observational data on the two student dependent 
variables (academic engagement and disruption/non-compliance). The training included three 1- 
2 hour sessions. Trainings included: (a) discussions of the context and data collector 
responsibilities, (b) overview of CICO, (c) reviewing operational definitions (i.e., examples and 
non-examples), and (d) practicing taking direct observational data using videos simulations 
within classroom environments. During practice video simulations, IOA was assessed and 
compared to observational data collected from the first author and data collectors. Any 
disagreements after video simulation were discussed until 100% agreement was obtained. 

IOA in the study was calculated by dividing the exact item agreements by the exact item 
disagreements plus agreements and then multiplying by 100%. IOA data were collected in 41 
observation periods: at least 25% of the observations for baseline (M = 35%) and intervention (M 
= 42.6%) phases for each student. IOA for academic engagement ranged between 78.3% - 100% 
(M = 92%) and for disruption/non-compliance ranged between 75% - 100% (M = 95.1%). When 
IOA was below 80%, data collectors met with the first author to review disagreements and 
operational definitions and come to agreement on definitions of academic engagement and 
disruption/non-compliance before the next observation. 

Treatment usability. To evaluate the usability and identify potential adaptations for the 
CICO-Secondary program, (a) strategies for adapting CICO to fit the social, organizational and 
developmental context of high school were developed with input from the CICO coordinator and 


high school administrator, and (b) end-of-study social validity questionnaires were administered 
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to the CICO coordinator, teachers and students. The social validity questionnaire for the CICO 
coordinator included seven questions, with six Likert-type rating questions (ranging from 1 = 
strongly disagree to 6 = strongly agree) and open-ended questions. Questions included asking the 
CICO coordinator: (a) whether CICO-Secondary improved students’ behaviors and (b) to 
provide feedback on components of CICO-Secondary that worked and could be improved (i.e., 
student selection criterion and process, student orientation to CICO-Secondary). Teacher and 
student social validity questionnaires consisted of seven questions (Likert-type rating questions 
and an open-ended question). Questions included: (a) whether the program improved student 
behaviors, (b) was easy to use, (c) if certain components would be better completed using an 
electronic application (i.e., student or teacher ratings), (d) whether teachers and students would 
participate in the program again and recommend it to others, and (e) advice on how to improve 
the program. 

CICO-Secondary 

Adaptations were made to the traditional CICO program in order to be more contextually 
and developmentally suitable for high schools and students. Adaptations to the traditional CICO 
program (Crone, Hawken, & Horner, 2010; Hawken & Horner, 2003) were as follows: 

Daily check-in and check-out. Although all of the students were trained to check-in 
each morning with the CICO coordinator, two of the students were given the option of picking 
up their daily point cards from their first period teachers to better match the logistics of their 
schedule. At the end of the school day, all students had the option of checking out with the CICO 
coordinator in person or dropping off their daily point card using a secure “mailbox” outside the 
CICO coordinator’s office. In this instance, the CICO coordinator would review the daily point 


cards and progress with students the following morning during check-in. 
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Daily point card. The daily point card was created with a bi-fold design that aimed to 
provide more privacy for students when handling the card. The front of the folded card included 
the school-wide expectations logo. The inside of the card included the five school-wide 
expectations and a place for teachers and students to rate whether these expectations were met 
for each period (see Figure 1). 

Self-regulation training. In addition to receiving the normal orientation to CICO 
procedures (check-in, check-out, use of daily point card), students were taught that CICO was 
about building their independence, success and self-regulation. This included explicit training in 
how students could self-monitor their behavior, and check-in with adults (e.g., get feedback and 
help) during the day. 

Ratings and feedback. Students were trained to use the daily point cards to self-assess 
whether they (a) greeted teachers at the beginning of each class and (b) followed the five school- 
wide expectations. By including a teacher and student response section on the daily point cards, 
students and teachers could each rate whether students met their daily goals. When there were 
disagreements between student and teacher scores, teachers had the opportunity to provide 
positive and corrective feedback. 

Home and school communication. Rather than having students take their daily point 
cards home to parents/guardians to review and sign, students were taught to leave their 
completed cards with the CICO coordinator (either in person during a check-out or in the 
designated secure “mailbox”). Home and school communication occurred instead through brief, 
weekly emails sent from the school administration indicating successes the student was having 
(e.g. “we are continuing to see improvements in Andrew’s academic engagement in the 


classroom” and “we look forward to seeing another good week of classroom participation”). 
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Procedures 

To evaluate the impact of CICO-Secondary on student academic engagement and 
disruption/non-compliant problem behaviors, we used a concurrent, single-case, multiple 
baseline design across student participants (Gast & Ledford, 2014). 

Baseline phase. Prior to baseline observations, the research team met with participating 
teachers to identify their classroom expectations. For example, Andrew’s teacher for College and 
Career Readiness did not allow students to wear hoodies, however, Andrew’s Science 9 teacher 
allowed students to wear hoodies but strongly enforced a “no cell phone use policy,” which was 
not enforced by other teachers. The research team also conducted a 30 min training with the 
CICO coordinator that included: (a) a conceptual overview and purpose of CICO-Secondary, (b) 
the CICO-Secondary standard routines and procedures, (c) data collection and entry, (d) family 
communication, (e) setting goals and utilizing acknowledgements, and (f) problem solving (e.g., 
if students forget to check-in pick up their daily point card). The training materials and content 
are available upon request from the first author. Baseline observations continued until there was 
a stable rate of responding for academic engagement and a minimum of five observations had 
occurred for a least one student (Kratochwill et al., 2010). 

Student and teacher training phase. Prior to intervention, classroom teachers were sent 
emails from the school administration informing them of the study. Research staff then followed 
up with individual teachers to invite them to participate. All teachers for the selected students 
agreed to participate. Following baseline data collection for each student, students and their 
teachers were trained on the CICO-Secondary program in a staggered fashion, as students 


entered intervention. 
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Student trainings included: (a) review of the daily CICO-Secondary routines and 
procedures, (b) teaching students how to self-assess whether they met their behavior goals (i.e., 
tallying points on daily point card), (c) how to obtain teacher feedback by approaching teachers 
in a positive manner, (d) how to accept positive and corrective feedback from teachers, and (e) 
how to problem solve (e.g., what happens if there is a substitute teacher). The research team 
trained the first student to model high training fidelity for the CICO-Secondary coordinator. The 
CICO coordinator then conducted the second student training with research staff observing and 
providing feedback. The remaining students were then trained by the CICO coordinator 
independently. The student trainings lasted approximately 15-20 min. All training materials and 
content for the students are available upon request from the first author. 

Training for classroom teachers consisted of: (a) conceptual overview and purpose of 
CICO-Secondary, (b) describing their roles and responsibilities (i.e., reviewing, rating, and 
signing the students’ daily point cards), and (c) how to give positive and constructive feedback to 
students (i.e., providing examples of positive and specific praise statements). Due to the small 
number of students in this study, teachers were trained individually. Research staff conducted 
these brief (10 min) trainings. If teachers had already been trained by the research team because 
a previous student had begun participating in CICO-Secondary, they only received an additional 
5 min booster training by the research team reviewing the critical components of CICO- 
Secondary. All training materials and content delivered to the teachers are also available upon 
request from the first author. 

CICO-Secondary phase. The CICO phase consisted of implementing CICO-Secondary. 
Since the school did not have a Tier 2 systems team or CICO team, and due to the small scope of 


the study, the research team worked directly with the CICO coordinator to implement the 
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program, monitor student and teacher fidelity, problem solve implementation issues, and review 
student fidelity and outcome data. Social validity questionnaires were administered to the CICO 
coordinator, teachers, and students within one week after the intervention phase. 
Procedural Fidelity 

Daily point cards were used to assess the extent that students participated in the daily 
components of CICO-Secondary. To obtain high procedural fidelity, students needed to (a) 
check-in with the CICO coordinator and collect their cards in the morning, (b) use their daily 
card to self-assess whether they greeted their teachers at the beginning of each class period, (c) 
use their daily card at the end of each class to self-assess if they met behavioral expectations, (d) 
obtain feedback from teachers on whether they greeted their teachers and if they followed 
classroom behavioral expectations, and (e) check-out at the end of the day with the CICO 
coordinator or return their cards to the CICO mailbox. 

Since there were four class periods each day, this presented students with a total of 14 
steps to complete the daily CICO-Secondary implementation cycle. However, if there was a 
substitute teacher, students were trained to only self-assess their behaviors and not obtain teacher 
feedback on whether they greeted or met their behavior goals. On those days, students completed 
one fewer implementation step for every substitute teacher. 
Analysis 

Descriptive statistics were used to assess implementation fidelity and social validity. 
Visual analysis of the observational data was used to determine whether a functional relation was 
found between the implementation of the program and improved student behaviors. Using What 


Works Clearinghouse single-case design standards (Kratochwill et al., 2010), we evaluated 
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visual analyses of the data in baseline and intervention phases based on data level, trend, 


variability, immediacy of effect, and overlapping data across phases. 


Results 


Fidelity of CICO-Secondary 

A central concern for the study was if CICO-Secondary could be implemented with high 
procedural fidelity. Information provided through direct observation, permanent product, and 
self-report from staff indicate high implementation of CICO-Secondary procedures by school 
personnel. Students were identified, oriented, and supported by the CICO coordinator and 
teachers as the program stipulates. There was lower procedural fidelity; however, when 
considering student behavior. CICO-Secondary can be expected to impact student behavior only 
if students engage in the procedures that will expose them to needed supports (i.e., morning 
check-in, greeting teachers, self-assessment throughout the day, recruiting teacher assessment, 
and check-out). Only one student, David, obtained his daily CICO point card, self-assessed, 
recruited teacher feedback and checked-out regularly, after completing the orientation (i.e. 9 of 
11 days of the intervention). Procedural fidelity for David ranged from 85.7% to 28.6%, and he 
was highly consistent with picking up his card, self-assessment, recruiting teacher feedback, and 
returning his card (78% to 100% of opportunities). He was less likely to greet teachers at the 
beginning of class (14% of opportunities). 

In contrast, Andrew only turned in a completed card three out of the 19 days of 
participation in CICO-Secondary, even with the adaptation of picking up his card at his first 
period class. Bethany turned in a completed card only two days (the 3™ and 4" days of 


intervention). Colin turned in a completed card on only one day (the first day of intervention), 
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and Ellise did not complete a card on either of the days she was on CICO-Secondary. In general, 
students were more likely to pick up their cards and self-assess and less likely to greet teachers 
upon entering class, obtain teacher feedback and turn in their cards. 

Impact of CICO-Secondary 

The percentage of disruption/non-compliance and academic engagement behaviors for 
the students, across baseline and intervention phases, are displayed in Figure 2. Based on visual 
analysis and poor procedural fidelity, there were no functional relations between the 
implementation of CICO-Secondary and decreases in student disruption/non-compliance or 
increases in academic engagement behaviors. We did, however, observe a basic effect when 
CICO-Secondary was implemented with fidelity for David, resulting in a distinct reduction in 
disruption/non-compliance, coupled with improved academic engagement. 

Disruption/non-compliance. Results for Andrew indicate a mean level change in 
disruption/non-compliance behaviors from 70.12% (range = 12.3 — 100%) in baseline to 44.11% 
(range = 0 — 100%) in CICO-Secondary. There were decreasing trends and high variability in the 
baseline and intervention phases, as well as high data overlap across phases. An immediacy of 
effect was observed when the program was implemented. 

For Bethany, there was a small mean level change in disruption/non-compliance from 
24.27% (range = 0 — 91.7%) in baseline to 22.18% (range = 0 — 88.3%) in intervention. An 
increasing trend in disruption/non-compliance in baseline and a decreasing trend in intervention 
was observed. There was moderate variability across phases and high data overlap across phases. 
An immediate increase in disruption/non-compliance behaviors was observed once the program 
was implemented. It is likely the timing of these problem behaviors was related to Bethany not 


taking her prescribed medication. The dashes in Figure 2 for Bethany represent two observations 
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on days the school administration noted she had refused to take her medication. On those days, 
data collectors noted that Bethany was lethargic (e.g., head on desk) and would struggle to 
participate with classroom activities. 

A small mean level change was observed in Colin’s disruption/non-compliance behaviors 
from 17.21% (range = 0 — 100%) in baseline to 12.66% (3.3 — 30%) in intervention. Decreasing 
trends in baseline and intervention phases were observed, and less variability in the intervention 
phase. There were high amounts of data overlap across phases and a small-to-moderate increase 
in disruption/non-compliance behaviors in intervention. 

Once CICO-Secondary was implemented for David, there was a mean level reduction in 
disruption/non-compliance behaviors from 20.70% (range = 0 — 90%) in baseline to 4.86% 
(range = 0 — 30%) in intervention. There was moderate variability in disruption/non-compliance 
behaviors in baseline. Once intervention started, there was a small, immediate decrease in 
disruption/non-compliance behaviors and a decreasing trend. There was little data variability in 
the CICO-Secondary phase and overlapping data across phases. 

Finally, for Ellise, there was small increase in mean level disruption/non-compliance 
behaviors from baseline 11.54% (range = 0 — 36.7%) to 13.34 (range = 5 — 21.67%). However, 
Ellise was only observed twice in the intervention phases and she did not participate in the daily 
CICO-secondary activities (i.e., checked-in with coordinator, picked up daily point card). An 
immediate decrease in disruption/non-compliance behaviors were observed once CICO- 
Secondary was implemented. In addition, in both baseline and intervention phases, there was an 
increasing trend in disruption/non-compliance behaviors and with small-to-moderate data 


variability and high data overlap across phases. 
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Academic engagement. For Andrew, results showed a mean level change in academic 
engagement from 32.48 (range = 1.7 — 53.3%) in baseline to 70.86% (range = 38 — 100%) in 
intervention. There was an increasing trend, with moderate variability in responding, across both 
baseline and intervention phases. After the program was implemented, there was an immediate 
increase in academic engagement, with minor data overlap across phases. 

For Bethany, mean level change in academic engagement went from 46.38% (range = 6.7 
— 100%) in baseline to 53.99% (range = 0 — 100%) in the intervention. There was a decreasing 
trend in baseline and an increasing trend in intervention, with high variability across phases. 
There was an immediate decrease in academic engagement when the program was implemented; 
however, it is likely that medication refusal also impacted academic engagement. 

For Colin, there was a mean level change in academic engagement from 49.83 (range = 0 
— 100%) in baseline to 63.68% (range = 13.3 — 93.3%) in the intervention phase. There was a 
decreasing trend in baseline and an increasing trend in the intervention; however, both phases 
included high variability and overlapping data. There was a large increase in academic 
engagement (immediacy of effect) once the program was implemented. 

After implementation of CICO-Secondary, David’s academic engagement increased from 
62.34% (range = 6.7 — 93.3%) in baseline to 79.7% (35 — 100%) in intervention. There was an 
increasing trend in baseline and intervention phases with moderate-to-high data variability across 
phases. 

Ellise’s mean level of academic engagement decreased from 68.81% (range = 23.3 — 
100%) in baseline to 55.85% (range = 31.7 — 80%). There was an initial increasing trend in 
academic engagement in baseline and a decreasing trend in the intervention phase, with 


moderate-to-high data variability across phases. 
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Usability of CICO-Secondary 

Social validity questionnaires from the CICO coordinator, classroom teachers, and 
students were used to evaluate the usability of CICO-Secondary. The CICO coordinator rated the 
student and teacher orientations to CICO-Secondary has having worked well (rated 5 on a 1 to 6 
scale) and the implementation of the program has having not work well (rated 3). Concerning the 
implementation process, the CICO coordinator discussed that it was “tricky to implement in 
spring. Need a person [CICO coordinator] with time to dedicate and access to student population 
participating.” The coordinator did not perceive CICO Secondary to have improved students’ 
behaviors (rated 3), but did somewhat agree that components of the program would be improved 
if completed on an electronic application (rated as 4). For example, the coordinator discussed 
that it is important to “maximize convenience for students [participating in CICO-Secondary] to 
increase participation.” 

Seventeen of the 24 teachers (M = 70.8%) completed the social validity surveys. Table 1 
includes a summary of the descriptive findings from the teacher survey. The highest rated item 
was that teachers would participate again in the program (M = 4.59, SD = 1.33) and the lowest 
rated item related to perceived improvements in students positive behaviors in class (M = 2.76, 
SD = 1.39). Qualitative responses on how the program could be improved included (a) selection 
of students for the program, (b) time of the year the program was implemented, and (c) including 
an online or electronic application component to the program to collect student and teacher 
ratings. To illustrate, one teacher said both of their students “exhibit a lack of 
attention/organization which [made] their ability to remember extra steps (get card, keep card, 
ask for rating) quite difficult.” Related to when CICO-Secondary was implemented, another 


teacher said “I did not see any significant change in the behavior of the students participating. By 
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the end of May they were very set in their ways. This may be more effective earlier in the 
process.” In regard to an online application component, one teacher said “using an app would 
improve the organization — they wouldn’t lose the card, but it would also be a gateway to getting 
districted by other notifications on their phone or using CICO as an excuse to be on their phone 
during class.” 

All students, except Ellise, completed the social validity survey. Items rated the highest 
included I would recommend CICO-Secondary for other students who may be struggling in 
school (M = 4.25, SD = 4.5) and CICO-Secondary helped improve my behavior at school (M = 
3.75, SD = 1.26) and It would be better if CICO-Secondary could be completed on an 
application instead of the paper card (M = 3.75, SD = 2.22). The item rated the lowest was It 
was easy to participate in the CICO-Secondary (M = 1.75, SD = 1.50). When asked how CICO- 
Secondary could be improved, three of the students suggested strategies for helping them 
remember to complete daily components of the program. For example, one of the students said 
“find another way to turn it [paper point card] in [be]cause I never remember” and another wrote 


“T forgot it a lot. It would be easier if you got like a notification on your phone.” 


Discussion 
The purpose of this preliminary study was to investigate the usability and impact of an 
adapted CICO program for high school students in one high school. Overall, we found that the 
CICO coordinator, classroom teachers, and the students rated the CICO-Secondary protocol as 
being useable. Unfortunately, the fidelity data indicated that while the staff implemented major 
elements of the program as intended, too often students did not receive sufficient support to 


perform the essential steps of the CICO cycle (i.e., did not complete morning check-in, did not 
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self-monitor, did not recruit teacher feedback). It is worth noting that David was the exception 
who did perform the CICO steps, and his data provide descriptive support for a basic effect 
between the implementation of the program and a decrease in disruption/non-compliance 
behaviors. 

Findings from this study provide support that CICO could have promise in high schools, 
especially when procedural components are adapted for the high school context and development 
level of the students. This pilot study sought to extent the survey findings by Kittelman et al. 
(2018) by adapting and testing procedural components CICO for high schools (e.g., varying 
check-in/check-out procedures, modifying daily point card). In addition, similar to Kittelman et 
al. (2018) and Swain-Bradway (2009), focusing on increasing academic behaviors (e.g., 
academic engagement, completing classwork) was an important area of emphasis for school 
personnel. In addition, based on the usability findings of this study, we found that the majority of 
high school personnel rated highly that they would participate in the program again, and the 
majority of students rated highly that they would recommend the program to other students who 
may be struggling in schools; however, further research on adapting and testing procedural 
components of CICO are warranted. 

Limitations and Recommendations for Future Research 

There were a number of limitations to this study. These included the low level of 
procedural fidelity by students, the variable adaptions that were made to CICO to meet students’ 
needs, and the time of year the study was implemented. First, a major limitation of this study was 
the low level of student participation in the daily CICO routines (i.e., check-in with CICO 
coordinator, complete and return daily point cards). The majority of the students inconsistently 


participated in the program. If students did not check-in at the beginning of the day, they were 
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not able to access other critical components (i.e., self-rating and getting feedback on school-wide 
behavior expectations). Although two students had options for the morning check-in (CICO 
Coordinator or first period teacher), this improved their initial use of the card early in the day, 
but did not improve their use of the card throughout the day (or turning in of the card). In 
addition, because some students had the option of checking in with the CICO coordinator or their 
first period teacher, this introduced a level of variability in the program across students, which 
can make comparison of the effectiveness of the program across students more difficult. The 
adaptions were made to increase the likelihood that students would follow-through with the 
procedural elements of the program. Future examination of ways to increase student fidelity of 
completing morning check-ins and afternoon check-outs is important to establishing conclusive 
evidence for the effectiveness of the program in future research. For example, future research 
may want to explore adapting other procedural components of the traditional CICO program 
(e.g., paper daily point card, increasing self-regulation strategies) to test whether these adaptation 
may increase student participation and compliance in the program. If the program does not have 
initial fidelity, and the student does not experience the planned contingencies, the likelihood of a 
functional effect is reduced. 

Additionally, the time of the school year when CICO-Secondary was piloted was 
problematic. As baseline observations began late in spring, many of the students’ daily routines 
had been established (e.g., classes, meeting with friends, arrival times to school), which likely 
made it difficult for students to change routines to check-in with the CICO-coordinator. 
Likewise, as students had already established relationships with their classroom teachers, 
students may have been more reluctant to want to greet and receive feedback on their behaviors 


from them, especially if students had developed negative relationships with some of their 
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teachers. Future research should consider training school personnel (CICO coordinator, Tier 2 
implementation team, and staff) to implement the program at the beginning of the school year to 
maximize school personnel and student buy-in and participation. 

Recommendations for Practice 

Role of CICO coordinator. As in lower grades, CICO-Secondary requires a coordinator 
to ensure procedural fidelity. This person should have 10 hours per week available to: train and 
support the teachers/students, conduct the daily CICO sessions with students, enter the data and 
attend meetings. A major role this person needs to fulfill is to train students and school personnel 
on the program and problem solve to ensure fidelity of implementation. Though the person 
fulfilling this coordinator role in this study had behavior expertise (i.e., behavior specialist for 
the school) she did not have the flexibility and protected time to complete the required tasks. For 
example, even if the school provided “training” during a faculty meeting, the coordinator still 
needed to have the time to train or re-orient teachers for students as they began the program. In 
addition, throughout the intervention phases, we observed times when students may have 
benefitted from booster training, or additional prompts, and the personnel time to complete these 
supports was unavailable. 

Daily check-in/check-out. Morning check-ins and afternoon check-outs are critical to 
the successful implementation of CICO-Secondary. Unfortunately, these times of day are social 
times in high school when students are highly engaged in social reinforcers with friends. Asking 
students to take time to check-in with the CICO coordinator, rather than visit with friends, 
proved challenging. Additional adaptations, such as electronic prompting (e.g., phone alarms, 
email) or companion check-ins (recruit a friend to help prompt) to remind students to check-in 


with the CICO coordinator in the momings may be useful. 
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Selection of students. Standard eligibility criteria that would remain the same for selection 
of high school students for CICO include: (a) referral from staff or parents, (b) engaging in 
problem behavior that warrants change but does not place the student at risk of alternative 
placement, and (c) not currently receiving intensive interventions for academic or problem 
behaviors. In addition, because of the academic focus in high school, schools might consider in 
determining eligibility criteria the display of low academic enabler skills (e.g., time management, 
homework completion) across multiple academic settings. The function of any absenteeism also 
needs to be understood to be sure it aligns with the function of the program. 

Consistency of expectations. Unlike the lower grades, students in this study had multiple 
teachers and common school-wide expectations were interpreted differently across classes. 
School-wide expectations, established as part of Tier 1 systems and implemented across 
classrooms, make the teaching of expected behaviors, and the acknowledgement for positive 
behaviors more predictable for students. Interestingly, student behaviors in this study were often 
consistent in the same classrooms, but varied across classrooms. An example of the effects of 
differing across classroom contingencies is available in Figure 3 for David. He had higher 
baseline rates of disruption/non-compliant behaviors in Period 2 compared to Period 1, which 
may have been directly related to the higher level of consistency and acknowledgement he 
experienced in Period 1. 

Conclusion 

This study provides several meaningful extensions to research in the area of CICO. We 
describe procedural adaptations to CICO that may improve use in high schools. Challenges 
related to CICO implementation fidelity and descriptive data suggesting promise of CICO for 


high school students at risk indicate the need for additional research on this topic. 
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Table 1 
Descriptive Social Validity Teacher Findings on the Usability of CICO-Secondary (1 = strongly disagree to 6 = strongly agree) 


CICO has helped CICO has helped It is easy to It would be better If I had a choice, I would 
improve my increase my participate in if CICO could be I would recommend 
students' students' positive CICO completed on an participate in CICO as a 
behavior in class behaviors in app instead of the CICO again potential 
class paper card intervention for 
other students 
who may be 
struggling in 
school 
Teacher 1 4 4 5 6 6 6 
Teacher 2 3 4 3 5 5 3 
Teacher 3 2 1 4 3 4 2 
Teacher 4 3 2 6 3 5 4 
Teacher 5 4 4 5 4 6 5 
Teacher 6 5 3 5 3 5 3 
Teacher 7 5 5 5 2 5 4 
Teacher 8 5 4 5 3 5 5 
Teacher 9 4 4 1 3 4 4 
Teacher 10 1 1 5 1 5 1 
Teacher 11 1 1 1 4 1 2 
Teacher 12 2 2 4 4 4 2 
Teacher 13 3 3 4 2 3 4 
Teacher 14 5 3 5 3 6 6 
Teacher 15 1 1 6 3 6 6 
Teacher 16 1 1 2 3 3 
Teacher 17 5 4 5 2 5 5 
M 3.18 2.77 4.18 3.19 4.59 3.82 
Mdn 3 3 5 3 5 4 
SD 1.59 1.39 1.55 1.22 1.33 1.55 


Note. M = Mean; Mdn = Median; SD = Standard deviation 
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SHARP Card 
Student Name: Date: 
0=Try Again 1 = Okay 2 = Great 
Period 1 or 2 Period 3 or 4 Period 5 or 6 Period 7 or 8 

Student | Teacher | Student | Teacher | Student | Teacher | Student | Teacher 
Greet Yor N/JY or N}|Y or N/|Y or NJ Y or N}|Y or N/}Y or NJY or N 
Self-aware 0122/0 12/0 12/0 12/0 12;0 1 2);0 12/0 1 2 
Honest 0122/0 12/0 12/0 12/0 12;0 1 2);0 12/0 1 2 
Appreciative |}O 1 2/0 12/0 12/0 12/0 12/0 12/0 12;0 1 2 
Respect 0 12/0 12/0 12/0 12/0 12;0 1 2);0 12/0 1 2 
Persistent 0 12/0 12/0 12/0 12/0 12;0 1 2);0 12/0 1 2 
Teacher 
Initials 
Feedback 
% Goal: Pts Possible: Pts Received: % of Pts Earned: Goal Met? Y or N 


Figure 1. The daily point card used by students participating in CICO-Secondary. 
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Figure 2. Percent of disruption/non-compliant (left) and academic engagement (right) behaviors 


across the five students. 


Note. Dashes in graph for Bethany indicate days she refused to take medication. 
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Figure 3. Differences in disruption/non-compliant behaviors across class periods for David. 


